{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "5525a271",
   "metadata": {},
   "source": [
    "# Recommendation System"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0223ff80",
   "metadata": {},
   "source": [
    "In this example we are creating a movie recommendation system by extractung feature vectors using PaddlePaddle, importing movie vectors into Milvus, and then searching in Milvus and Redis."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6b4edf8a",
   "metadata": {},
   "source": [
    "## Data\n",
    "In this project, we use [MovieLens 1M](https://grouplens.org/datasets/movielens/1m/). This dataset contains 1,000,209 anonymous ratings of approximately 3,900 movies made by 6,040 MovieLens users. \n",
    "\n",
    "We use the following files:\n",
    "- movies.dat: Contains movie information.\n",
    "- movie_vectors.txt: Contains movie vectors that can be imported to Milvus easily.\n",
    "\n",
    "File structure:\n",
    "\n",
    " - movMovieID::Title::Genres   \n",
    "\n",
    "     - Titles are identical to titles provided by the IMDB (includingyear of release)\n",
    " \n",
    "     - Genres are pipe-separated\n",
    "\n",
    "     - Some MovieIDs do not correspond to a movie due to accidental duplicate entries and/or test entries\n",
    " \n",
    "    - Movies are mostly entered by hand, so errors and inconsistencies may exist\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ceedfea",
   "metadata": {},
   "source": [
    "## Requirements"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3dfeb422",
   "metadata": {},
   "source": [
    "Due to package constraints, this notebook needs to be run using Python 3.6/3.7 . It is recommended that you use a virtual enviroment like Conda, instructions for installing Conda can be found [here](https://conda.io/projects/conda/en/latest/user-guide/install/index.html). \n",
    "\n",
    "Currently, there is a dirty workaround that you can use. When installing the `requirements.txt`, pip will fail to install`sentencepiece`. If you rerun the notebook after the install fails and avoid redownloading the packages, the rest of the notebook should run without any hiccups.\n",
    "\n",
    "|  Packages |  Servers |\n",
    "| --------------- | -------------- |\n",
    "| pymilvus        | milvus-1.1.0   |\n",
    "| redis           | redis          |\n",
    "| paddle_serving_app |\n",
    "| paddlepaddle |\n",
    "\n",
    "\n",
    "We have included a requirements.txt file in order to easily satisfy the required packages. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af42466a-af8f-4f8b-b300-8f97e10b3617",
   "metadata": {},
   "source": [
    "## Up and Running"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b436afa7",
   "metadata": {},
   "source": [
    "### Installing Packages\n",
    "Install the required python packages with `requirements.txt`. If using Python 3.8, look at workaround in the Requirements section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "8d30a0db",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: pymilvus==1.1.0 in /Users/filiphaltmayer/opt/miniconda3/envs/coinbase_demo/lib/python3.8/site-packages (from -r requirements.txt (line 1)) (1.1.0)\n",
      "Requirement already satisfied: sentencepiece in /Users/filiphaltmayer/opt/miniconda3/envs/coinbase_demo/lib/python3.8/site-packages (from -r requirements.txt (line 2)) (0.1.91)\n",
      "Collecting paddle_serving_app==0.3.1\n",
      "  Using cached paddle_serving_app-0.3.1-py3-none-any.whl (52 kB)\n",
      "Requirement already satisfied: paddlepaddle in /Users/filiphaltmayer/opt/miniconda3/envs/coinbase_demo/lib/python3.8/site-packages (from -r requirements.txt (line 4)) (2.1.0)\n",
      "Requirement already satisfied: redis in /Users/filiphaltmayer/opt/miniconda3/envs/coinbase_demo/lib/python3.8/site-packages (from -r requirements.txt (line 5)) (3.5.3)\n",
      "Requirement already satisfied: requests>=2.22.0 in /Users/filiphaltmayer/opt/miniconda3/envs/coinbase_demo/lib/python3.8/site-packages (from pymilvus==1.1.0->-r requirements.txt (line 1)) (2.25.1)\n",
      "Requirement already satisfied: grpcio>=1.22.0 in /Users/filiphaltmayer/opt/miniconda3/envs/coinbase_demo/lib/python3.8/site-packages (from pymilvus==1.1.0->-r requirements.txt (line 1)) (1.37.0)\n",
      "Requirement already satisfied: grpcio-tools>=1.22.0 in /Users/filiphaltmayer/opt/miniconda3/envs/coinbase_demo/lib/python3.8/site-packages (from pymilvus==1.1.0->-r requirements.txt (line 1)) (1.37.0)\n",
      "Requirement already satisfied: ujson>=2.0.0 in /Users/filiphaltmayer/opt/miniconda3/envs/coinbase_demo/lib/python3.8/site-packages (from pymilvus==1.1.0->-r requirements.txt (line 1)) (4.0.2)\n",
      "Requirement already satisfied: opencv-python<=4.2.0.32 in /Users/filiphaltmayer/opt/miniconda3/envs/coinbase_demo/lib/python3.8/site-packages (from paddle_serving_app==0.3.1->-r requirements.txt (line 3)) (4.2.0.32)\n",
      "Collecting shapely\n",
      "  Using cached Shapely-1.7.1-cp38-cp38-macosx_10_9_x86_64.whl (1.0 MB)\n",
      "Requirement already satisfied: pillow in /Users/filiphaltmayer/opt/miniconda3/envs/coinbase_demo/lib/python3.8/site-packages (from paddle_serving_app==0.3.1->-r requirements.txt (line 3)) (8.2.0)\n",
      "Collecting sentencepiece\n",
      "  Using cached sentencepiece-0.1.83.tar.gz (497 kB)\n",
      "\u001b[31m    ERROR: Command errored out with exit status 1:\n",
      "     command: /Users/filiphaltmayer/opt/miniconda3/envs/coinbase_demo/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '\"'\"'/private/var/folders/t1/4dsdtlv12r7cpxx9hkz33b840000gn/T/pip-install-pj4cikfj/sentencepiece_83891e74c0de41d08e7b24394a6852ce/setup.py'\"'\"'; __file__='\"'\"'/private/var/folders/t1/4dsdtlv12r7cpxx9hkz33b840000gn/T/pip-install-pj4cikfj/sentencepiece_83891e74c0de41d08e7b24394a6852ce/setup.py'\"'\"';f = getattr(tokenize, '\"'\"'open'\"'\"', open)(__file__) if os.path.exists(__file__) else io.StringIO('\"'\"'from setuptools import setup; setup()'\"'\"');code = f.read().replace('\"'\"'\\r\\n'\"'\"', '\"'\"'\\n'\"'\"');f.close();exec(compile(code, __file__, '\"'\"'exec'\"'\"'))' egg_info --egg-base /private/var/folders/t1/4dsdtlv12r7cpxx9hkz33b840000gn/T/pip-pip-egg-info-k1feq6b_\n",
      "         cwd: /private/var/folders/t1/4dsdtlv12r7cpxx9hkz33b840000gn/T/pip-install-pj4cikfj/sentencepiece_83891e74c0de41d08e7b24394a6852ce/\n",
      "    Complete output (7 lines):\n",
      "    Traceback (most recent call last):\n",
      "      File \"<string>\", line 1, in <module>\n",
      "      File \"/private/var/folders/t1/4dsdtlv12r7cpxx9hkz33b840000gn/T/pip-install-pj4cikfj/sentencepiece_83891e74c0de41d08e7b24394a6852ce/setup.py\", line 29, in <module>\n",
      "        with codecs.open(os.path.join('..', 'VERSION'), 'r', 'utf-8') as f:\n",
      "      File \"/Users/filiphaltmayer/opt/miniconda3/envs/coinbase_demo/lib/python3.8/codecs.py\", line 905, in open\n",
      "        file = builtins.open(filename, mode, buffering)\n",
      "    FileNotFoundError: [Errno 2] No such file or directory: '../VERSION'\n",
      "    ----------------------------------------\u001b[0m\n",
      "\u001b[33mWARNING: Discarding https://files.pythonhosted.org/packages/1b/87/c3c2fa8cbec61fffe031ca9f0da512747520bec9be7f886f748457daac31/sentencepiece-0.1.83.tar.gz#sha256=d194cf7431dd87798963ff998380f1c02ff0f9e380cc922a07926b69e21c4e2b (from https://pypi.org/simple/sentencepiece/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.\u001b[0m\n",
      "  Using cached sentencepiece-0.1.3.tar.gz (498 kB)\n",
      "\u001b[31m    ERROR: Command errored out with exit status 1:\n",
      "     command: /Users/filiphaltmayer/opt/miniconda3/envs/coinbase_demo/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '\"'\"'/private/var/folders/t1/4dsdtlv12r7cpxx9hkz33b840000gn/T/pip-install-pj4cikfj/sentencepiece_23eab281a1194adc92bfe5cd817d04e0/setup.py'\"'\"'; __file__='\"'\"'/private/var/folders/t1/4dsdtlv12r7cpxx9hkz33b840000gn/T/pip-install-pj4cikfj/sentencepiece_23eab281a1194adc92bfe5cd817d04e0/setup.py'\"'\"';f = getattr(tokenize, '\"'\"'open'\"'\"', open)(__file__) if os.path.exists(__file__) else io.StringIO('\"'\"'from setuptools import setup; setup()'\"'\"');code = f.read().replace('\"'\"'\\r\\n'\"'\"', '\"'\"'\\n'\"'\"');f.close();exec(compile(code, __file__, '\"'\"'exec'\"'\"'))' egg_info --egg-base /private/var/folders/t1/4dsdtlv12r7cpxx9hkz33b840000gn/T/pip-pip-egg-info-vqblob8o\n",
      "         cwd: /private/var/folders/t1/4dsdtlv12r7cpxx9hkz33b840000gn/T/pip-install-pj4cikfj/sentencepiece_23eab281a1194adc92bfe5cd817d04e0/\n",
      "    Complete output (5 lines):\n",
      "    Package sentencepiece was not found in the pkg-config search path.\n",
      "    Perhaps you should add the directory containing `sentencepiece.pc'\n",
      "    to the PKG_CONFIG_PATH environment variable\n",
      "    No package 'sentencepiece' found\n",
      "    Failed to find sentencepiece pkgconfig\n",
      "    ----------------------------------------\u001b[0m\n",
      "\u001b[33mWARNING: Discarding https://files.pythonhosted.org/packages/fd/45/6d0eb609d5cd81df094aab71a867b2ab6b315ffd592e78fb94a625c4d6aa/sentencepiece-0.1.3.tar.gz#sha256=bfdfa5699ed8c44c55c9e19c46327bb9b3106828fce631db10e83b29a10bc315 (from https://pypi.org/simple/sentencepiece/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.\u001b[0m\n",
      "  Using cached sentencepiece-0.1.2.tar.gz (496 kB)\n",
      "\u001b[31m    ERROR: Command errored out with exit status 1:\n",
      "     command: /Users/filiphaltmayer/opt/miniconda3/envs/coinbase_demo/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '\"'\"'/private/var/folders/t1/4dsdtlv12r7cpxx9hkz33b840000gn/T/pip-install-pj4cikfj/sentencepiece_8498191050f84ee3bda6572486fa2c2f/setup.py'\"'\"'; __file__='\"'\"'/private/var/folders/t1/4dsdtlv12r7cpxx9hkz33b840000gn/T/pip-install-pj4cikfj/sentencepiece_8498191050f84ee3bda6572486fa2c2f/setup.py'\"'\"';f = getattr(tokenize, '\"'\"'open'\"'\"', open)(__file__) if os.path.exists(__file__) else io.StringIO('\"'\"'from setuptools import setup; setup()'\"'\"');code = f.read().replace('\"'\"'\\r\\n'\"'\"', '\"'\"'\\n'\"'\"');f.close();exec(compile(code, __file__, '\"'\"'exec'\"'\"'))' egg_info --egg-base /private/var/folders/t1/4dsdtlv12r7cpxx9hkz33b840000gn/T/pip-pip-egg-info-299u97b6\n",
      "         cwd: /private/var/folders/t1/4dsdtlv12r7cpxx9hkz33b840000gn/T/pip-install-pj4cikfj/sentencepiece_8498191050f84ee3bda6572486fa2c2f/\n",
      "    Complete output (5 lines):\n",
      "    Package sentencepiece was not found in the pkg-config search path.\n",
      "    Perhaps you should add the directory containing `sentencepiece.pc'\n",
      "    to the PKG_CONFIG_PATH environment variable\n",
      "    No package 'sentencepiece' found\n",
      "    Failed to find sentencepiece pkgconfig\n",
      "    ----------------------------------------\u001b[0m\n",
      "\u001b[33mWARNING: Discarding https://files.pythonhosted.org/packages/db/85/d618ae3089a332f5e975820abee2de54e4e2ecbec99b08d80cb6ab8f5b59/sentencepiece-0.1.2.tar.gz#sha256=0e87925ab14aab269694530d47d3c9b68d67a75a1b6f0cc5ec4f631c6784fc58 (from https://pypi.org/simple/sentencepiece/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.\u001b[0m\n",
      "  Using cached sentencepiece-0.1.1.tar.gz (496 kB)\n",
      "\u001b[31m    ERROR: Command errored out with exit status 1:\n",
      "     command: /Users/filiphaltmayer/opt/miniconda3/envs/coinbase_demo/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '\"'\"'/private/var/folders/t1/4dsdtlv12r7cpxx9hkz33b840000gn/T/pip-install-pj4cikfj/sentencepiece_041c95fed6fb41a3af214b21455ca4c9/setup.py'\"'\"'; __file__='\"'\"'/private/var/folders/t1/4dsdtlv12r7cpxx9hkz33b840000gn/T/pip-install-pj4cikfj/sentencepiece_041c95fed6fb41a3af214b21455ca4c9/setup.py'\"'\"';f = getattr(tokenize, '\"'\"'open'\"'\"', open)(__file__) if os.path.exists(__file__) else io.StringIO('\"'\"'from setuptools import setup; setup()'\"'\"');code = f.read().replace('\"'\"'\\r\\n'\"'\"', '\"'\"'\\n'\"'\"');f.close();exec(compile(code, __file__, '\"'\"'exec'\"'\"'))' egg_info --egg-base /private/var/folders/t1/4dsdtlv12r7cpxx9hkz33b840000gn/T/pip-pip-egg-info-1i1g8y1u\n",
      "         cwd: /private/var/folders/t1/4dsdtlv12r7cpxx9hkz33b840000gn/T/pip-install-pj4cikfj/sentencepiece_041c95fed6fb41a3af214b21455ca4c9/\n",
      "    Complete output (5 lines):\n",
      "    Package sentencepiece was not found in the pkg-config search path.\n",
      "    Perhaps you should add the directory containing `sentencepiece.pc'\n",
      "    to the PKG_CONFIG_PATH environment variable\n",
      "    No package 'sentencepiece' found\n",
      "    Failed to find sentencepiece pkgconfig\n",
      "    ----------------------------------------\u001b[0m\n",
      "\u001b[33mWARNING: Discarding https://files.pythonhosted.org/packages/38/53/5f0b4d0d0864c4438bb2349c9325acc9b4a72e72006f8ae3e4ea48ce3883/sentencepiece-0.1.1.tar.gz#sha256=b3a22bbbf8863c95447548f4b5b1288eae2acc6e47cfd61adc27d20ae959dcce (from https://pypi.org/simple/sentencepiece/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.\u001b[0m\n",
      "  Using cached sentencepiece-0.1.0.tar.gz (492 kB)\n",
      "\u001b[31m    ERROR: Command errored out with exit status 1:\n",
      "     command: /Users/filiphaltmayer/opt/miniconda3/envs/coinbase_demo/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '\"'\"'/private/var/folders/t1/4dsdtlv12r7cpxx9hkz33b840000gn/T/pip-install-pj4cikfj/sentencepiece_efc5edefe42644a894a312e4c71c26f8/setup.py'\"'\"'; __file__='\"'\"'/private/var/folders/t1/4dsdtlv12r7cpxx9hkz33b840000gn/T/pip-install-pj4cikfj/sentencepiece_efc5edefe42644a894a312e4c71c26f8/setup.py'\"'\"';f = getattr(tokenize, '\"'\"'open'\"'\"', open)(__file__) if os.path.exists(__file__) else io.StringIO('\"'\"'from setuptools import setup; setup()'\"'\"');code = f.read().replace('\"'\"'\\r\\n'\"'\"', '\"'\"'\\n'\"'\"');f.close();exec(compile(code, __file__, '\"'\"'exec'\"'\"'))' egg_info --egg-base /private/var/folders/t1/4dsdtlv12r7cpxx9hkz33b840000gn/T/pip-pip-egg-info-xw3ij_9m\n",
      "         cwd: /private/var/folders/t1/4dsdtlv12r7cpxx9hkz33b840000gn/T/pip-install-pj4cikfj/sentencepiece_efc5edefe42644a894a312e4c71c26f8/\n",
      "    Complete output (5 lines):\n",
      "    Package sentencepiece was not found in the pkg-config search path.\n",
      "    Perhaps you should add the directory containing `sentencepiece.pc'\n",
      "    to the PKG_CONFIG_PATH environment variable\n",
      "    No package 'sentencepiece' found\n",
      "    Failed to find sentencepiece pkgconfig\n",
      "    ----------------------------------------\u001b[0m\n",
      "\u001b[33mWARNING: Discarding https://files.pythonhosted.org/packages/1b/26/a721ab368de4d3e79ebcba905d3d407261898a95cdbd0790e4f8cabe5958/sentencepiece-0.1.0.tar.gz#sha256=6b76ff3598f729cf0064fd692431039110990572ee05d94c9ffc596d5d01dd05 (from https://pypi.org/simple/sentencepiece/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.\u001b[0m\n",
      "  Using cached sentencepiece-0.0.9.tar.gz (491 kB)\n",
      "\u001b[31m    ERROR: Command errored out with exit status 1:\n",
      "     command: /Users/filiphaltmayer/opt/miniconda3/envs/coinbase_demo/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '\"'\"'/private/var/folders/t1/4dsdtlv12r7cpxx9hkz33b840000gn/T/pip-install-pj4cikfj/sentencepiece_1a7b304ec2f74ccbb3c7991093678964/setup.py'\"'\"'; __file__='\"'\"'/private/var/folders/t1/4dsdtlv12r7cpxx9hkz33b840000gn/T/pip-install-pj4cikfj/sentencepiece_1a7b304ec2f74ccbb3c7991093678964/setup.py'\"'\"';f = getattr(tokenize, '\"'\"'open'\"'\"', open)(__file__) if os.path.exists(__file__) else io.StringIO('\"'\"'from setuptools import setup; setup()'\"'\"');code = f.read().replace('\"'\"'\\r\\n'\"'\"', '\"'\"'\\n'\"'\"');f.close();exec(compile(code, __file__, '\"'\"'exec'\"'\"'))' egg_info --egg-base /private/var/folders/t1/4dsdtlv12r7cpxx9hkz33b840000gn/T/pip-pip-egg-info-9jvkaxxf\n",
      "         cwd: /private/var/folders/t1/4dsdtlv12r7cpxx9hkz33b840000gn/T/pip-install-pj4cikfj/sentencepiece_1a7b304ec2f74ccbb3c7991093678964/\n",
      "    Complete output (5 lines):\n",
      "    Package sentencepiece was not found in the pkg-config search path.\n",
      "    Perhaps you should add the directory containing `sentencepiece.pc'\n",
      "    to the PKG_CONFIG_PATH environment variable\n",
      "    No package 'sentencepiece' found\n",
      "    Failed to find sentencepiece pkgconfig\n",
      "    ----------------------------------------\u001b[0m\n",
      "\u001b[33mWARNING: Discarding https://files.pythonhosted.org/packages/a3/06/add980badde94dd2ec5aa8058fb762946dd7db62aa3328c15186bcadaae0/sentencepiece-0.0.9.tar.gz#sha256=134f128a6903f3c7d7c0f31945803173b502db76e1b992d735f26c2ed158ef26 (from https://pypi.org/simple/sentencepiece/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.\u001b[0m\n",
      "  Using cached sentencepiece-0.0.7.tar.gz (493 kB)\n",
      "\u001b[31m    ERROR: Command errored out with exit status 1:\n",
      "     command: /Users/filiphaltmayer/opt/miniconda3/envs/coinbase_demo/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '\"'\"'/private/var/folders/t1/4dsdtlv12r7cpxx9hkz33b840000gn/T/pip-install-pj4cikfj/sentencepiece_4113bef85e7c4e12a8e5908f46fd4f43/setup.py'\"'\"'; __file__='\"'\"'/private/var/folders/t1/4dsdtlv12r7cpxx9hkz33b840000gn/T/pip-install-pj4cikfj/sentencepiece_4113bef85e7c4e12a8e5908f46fd4f43/setup.py'\"'\"';f = getattr(tokenize, '\"'\"'open'\"'\"', open)(__file__) if os.path.exists(__file__) else io.StringIO('\"'\"'from setuptools import setup; setup()'\"'\"');code = f.read().replace('\"'\"'\\r\\n'\"'\"', '\"'\"'\\n'\"'\"');f.close();exec(compile(code, __file__, '\"'\"'exec'\"'\"'))' egg_info --egg-base /private/var/folders/t1/4dsdtlv12r7cpxx9hkz33b840000gn/T/pip-pip-egg-info-nm5zf9k8\n",
      "         cwd: /private/var/folders/t1/4dsdtlv12r7cpxx9hkz33b840000gn/T/pip-install-pj4cikfj/sentencepiece_4113bef85e7c4e12a8e5908f46fd4f43/\n",
      "    Complete output (5 lines):\n",
      "    Package sentencepiece was not found in the pkg-config search path.\n",
      "    Perhaps you should add the directory containing `sentencepiece.pc'\n",
      "    to the PKG_CONFIG_PATH environment variable\n",
      "    No package 'sentencepiece' found\n",
      "    Failed to find sentencepiece pkgconfig\n",
      "    ----------------------------------------\u001b[0m\n",
      "\u001b[33mWARNING: Discarding https://files.pythonhosted.org/packages/ef/ba/17c0c4f8ccc746b2182c7e3c8292be0bdb37fbadeaf467d2f69565160764/sentencepiece-0.0.7.tar.gz#sha256=83a40cecb39a446176809dc53361d2efce93e3009cea0b5dc3ace3ab2902cde0 (from https://pypi.org/simple/sentencepiece/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.\u001b[0m\n",
      "  Using cached sentencepiece-0.0.6.tar.gz (492 kB)\n",
      "\u001b[31m    ERROR: Command errored out with exit status 1:\n",
      "     command: /Users/filiphaltmayer/opt/miniconda3/envs/coinbase_demo/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '\"'\"'/private/var/folders/t1/4dsdtlv12r7cpxx9hkz33b840000gn/T/pip-install-pj4cikfj/sentencepiece_0cbf26b538ff49ecb65896ae6c38b82c/setup.py'\"'\"'; __file__='\"'\"'/private/var/folders/t1/4dsdtlv12r7cpxx9hkz33b840000gn/T/pip-install-pj4cikfj/sentencepiece_0cbf26b538ff49ecb65896ae6c38b82c/setup.py'\"'\"';f = getattr(tokenize, '\"'\"'open'\"'\"', open)(__file__) if os.path.exists(__file__) else io.StringIO('\"'\"'from setuptools import setup; setup()'\"'\"');code = f.read().replace('\"'\"'\\r\\n'\"'\"', '\"'\"'\\n'\"'\"');f.close();exec(compile(code, __file__, '\"'\"'exec'\"'\"'))' egg_info --egg-base /private/var/folders/t1/4dsdtlv12r7cpxx9hkz33b840000gn/T/pip-pip-egg-info-vou13w_f\n",
      "         cwd: /private/var/folders/t1/4dsdtlv12r7cpxx9hkz33b840000gn/T/pip-install-pj4cikfj/sentencepiece_0cbf26b538ff49ecb65896ae6c38b82c/\n",
      "    Complete output (5 lines):\n",
      "    Package sentencepiece was not found in the pkg-config search path.\n",
      "    Perhaps you should add the directory containing `sentencepiece.pc'\n",
      "    to the PKG_CONFIG_PATH environment variable\n",
      "    No package 'sentencepiece' found\n",
      "    Failed to find sentencepiece pkgconfig\n",
      "    ----------------------------------------\u001b[0m\n",
      "\u001b[33mWARNING: Discarding https://files.pythonhosted.org/packages/8a/1e/5eaa80c2a01d0d40face64d7de2d525c9d77d6ffb15f74377eb373552d7a/sentencepiece-0.0.6.tar.gz#sha256=d101a584732789a2c1b3e633a0b1a8e367c86d2a607c5b911080e88f50483f4d (from https://pypi.org/simple/sentencepiece/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.\u001b[0m\n",
      "  Using cached sentencepiece-0.0.5.tar.gz (492 kB)\n",
      "\u001b[31m    ERROR: Command errored out with exit status 1:\n",
      "     command: /Users/filiphaltmayer/opt/miniconda3/envs/coinbase_demo/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '\"'\"'/private/var/folders/t1/4dsdtlv12r7cpxx9hkz33b840000gn/T/pip-install-pj4cikfj/sentencepiece_69561ff0519647d58ab60b3406e20529/setup.py'\"'\"'; __file__='\"'\"'/private/var/folders/t1/4dsdtlv12r7cpxx9hkz33b840000gn/T/pip-install-pj4cikfj/sentencepiece_69561ff0519647d58ab60b3406e20529/setup.py'\"'\"';f = getattr(tokenize, '\"'\"'open'\"'\"', open)(__file__) if os.path.exists(__file__) else io.StringIO('\"'\"'from setuptools import setup; setup()'\"'\"');code = f.read().replace('\"'\"'\\r\\n'\"'\"', '\"'\"'\\n'\"'\"');f.close();exec(compile(code, __file__, '\"'\"'exec'\"'\"'))' egg_info --egg-base /private/var/folders/t1/4dsdtlv12r7cpxx9hkz33b840000gn/T/pip-pip-egg-info-dyd3_jgf\n",
      "         cwd: /private/var/folders/t1/4dsdtlv12r7cpxx9hkz33b840000gn/T/pip-install-pj4cikfj/sentencepiece_69561ff0519647d58ab60b3406e20529/\n",
      "    Complete output (5 lines):\n",
      "    Package sentencepiece was not found in the pkg-config search path.\n",
      "    Perhaps you should add the directory containing `sentencepiece.pc'\n",
      "    to the PKG_CONFIG_PATH environment variable\n",
      "    No package 'sentencepiece' found\n",
      "    Failed to find sentencepiece pkgconfig\n",
      "    ----------------------------------------\u001b[0m\n",
      "\u001b[33mWARNING: Discarding https://files.pythonhosted.org/packages/9d/e6/ab32fac74b278dcde7e3f2310afea53c13f6e86fbc80eeb0e100c9038d65/sentencepiece-0.0.5.tar.gz#sha256=c3efbfc9598b1afb5d58d9f34f70f670838f455603357f63b096d5c71337ef85 (from https://pypi.org/simple/sentencepiece/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.\u001b[0m\n",
      "  Using cached sentencepiece-0.0.4.tar.gz (490 kB)\n",
      "\u001b[31m    ERROR: Command errored out with exit status 1:\n",
      "     command: /Users/filiphaltmayer/opt/miniconda3/envs/coinbase_demo/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '\"'\"'/private/var/folders/t1/4dsdtlv12r7cpxx9hkz33b840000gn/T/pip-install-pj4cikfj/sentencepiece_4400692862314717a120758840a0a26e/setup.py'\"'\"'; __file__='\"'\"'/private/var/folders/t1/4dsdtlv12r7cpxx9hkz33b840000gn/T/pip-install-pj4cikfj/sentencepiece_4400692862314717a120758840a0a26e/setup.py'\"'\"';f = getattr(tokenize, '\"'\"'open'\"'\"', open)(__file__) if os.path.exists(__file__) else io.StringIO('\"'\"'from setuptools import setup; setup()'\"'\"');code = f.read().replace('\"'\"'\\r\\n'\"'\"', '\"'\"'\\n'\"'\"');f.close();exec(compile(code, __file__, '\"'\"'exec'\"'\"'))' egg_info --egg-base /private/var/folders/t1/4dsdtlv12r7cpxx9hkz33b840000gn/T/pip-pip-egg-info-4h13drd_\n",
      "         cwd: /private/var/folders/t1/4dsdtlv12r7cpxx9hkz33b840000gn/T/pip-install-pj4cikfj/sentencepiece_4400692862314717a120758840a0a26e/\n",
      "    Complete output (5 lines):\n",
      "    Package sentencepiece was not found in the pkg-config search path.\n",
      "    Perhaps you should add the directory containing `sentencepiece.pc'\n",
      "    to the PKG_CONFIG_PATH environment variable\n",
      "    No package 'sentencepiece' found\n",
      "    Failed to find sentencepiece pkgconfig\n",
      "    ----------------------------------------\u001b[0m\n",
      "\u001b[33mWARNING: Discarding https://files.pythonhosted.org/packages/40/19/1878352c1adfc1df018bc69c962259774811ac0a90b2be3bc89b9907df81/sentencepiece-0.0.4.tar.gz#sha256=6896b6b43197cf3f4801757f4b8ca178251d223d38a1d857c714c040557c7608 (from https://pypi.org/simple/sentencepiece/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.\u001b[0m\n",
      "  Using cached sentencepiece-0.0.3.tar.gz (380 kB)\n",
      "\u001b[31m    ERROR: Command errored out with exit status 1:\n",
      "     command: /Users/filiphaltmayer/opt/miniconda3/envs/coinbase_demo/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '\"'\"'/private/var/folders/t1/4dsdtlv12r7cpxx9hkz33b840000gn/T/pip-install-pj4cikfj/sentencepiece_cc7d294a64f142d3b6656fa7a8678bb3/setup.py'\"'\"'; __file__='\"'\"'/private/var/folders/t1/4dsdtlv12r7cpxx9hkz33b840000gn/T/pip-install-pj4cikfj/sentencepiece_cc7d294a64f142d3b6656fa7a8678bb3/setup.py'\"'\"';f = getattr(tokenize, '\"'\"'open'\"'\"', open)(__file__) if os.path.exists(__file__) else io.StringIO('\"'\"'from setuptools import setup; setup()'\"'\"');code = f.read().replace('\"'\"'\\r\\n'\"'\"', '\"'\"'\\n'\"'\"');f.close();exec(compile(code, __file__, '\"'\"'exec'\"'\"'))' egg_info --egg-base /private/var/folders/t1/4dsdtlv12r7cpxx9hkz33b840000gn/T/pip-pip-egg-info-ir3qvasf\n",
      "         cwd: /private/var/folders/t1/4dsdtlv12r7cpxx9hkz33b840000gn/T/pip-install-pj4cikfj/sentencepiece_cc7d294a64f142d3b6656fa7a8678bb3/\n",
      "    Complete output (5 lines):\n",
      "    Package sentencepiece was not found in the pkg-config search path.\n",
      "    Perhaps you should add the directory containing `sentencepiece.pc'\n",
      "    to the PKG_CONFIG_PATH environment variable\n",
      "    No package 'sentencepiece' found\n",
      "    Failed to find sentencepiece pkgconfig\n",
      "    ----------------------------------------\u001b[0m\n",
      "\u001b[33mWARNING: Discarding https://files.pythonhosted.org/packages/96/ad/b1532b8bb8f20a392189de09d93eb4c7c95ccf98c93b55e9ab00db1bc61f/sentencepiece-0.0.3.tar.gz#sha256=7dbefd4ba670d5862e7925bf11792d535074dc06af8015558c6642648b770358 (from https://pypi.org/simple/sentencepiece/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.\u001b[0m\n",
      "  Using cached sentencepiece-0.0.2.tar.gz (378 kB)\n",
      "\u001b[31m    ERROR: Command errored out with exit status 1:\n",
      "     command: /Users/filiphaltmayer/opt/miniconda3/envs/coinbase_demo/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '\"'\"'/private/var/folders/t1/4dsdtlv12r7cpxx9hkz33b840000gn/T/pip-install-pj4cikfj/sentencepiece_2365cdc1bb284eee86c85d4b34007621/setup.py'\"'\"'; __file__='\"'\"'/private/var/folders/t1/4dsdtlv12r7cpxx9hkz33b840000gn/T/pip-install-pj4cikfj/sentencepiece_2365cdc1bb284eee86c85d4b34007621/setup.py'\"'\"';f = getattr(tokenize, '\"'\"'open'\"'\"', open)(__file__) if os.path.exists(__file__) else io.StringIO('\"'\"'from setuptools import setup; setup()'\"'\"');code = f.read().replace('\"'\"'\\r\\n'\"'\"', '\"'\"'\\n'\"'\"');f.close();exec(compile(code, __file__, '\"'\"'exec'\"'\"'))' egg_info --egg-base /private/var/folders/t1/4dsdtlv12r7cpxx9hkz33b840000gn/T/pip-pip-egg-info-oxiqbvx1\n",
      "         cwd: /private/var/folders/t1/4dsdtlv12r7cpxx9hkz33b840000gn/T/pip-install-pj4cikfj/sentencepiece_2365cdc1bb284eee86c85d4b34007621/\n",
      "    Complete output (5 lines):\n",
      "    Package sentencepiece was not found in the pkg-config search path.\n",
      "    Perhaps you should add the directory containing `sentencepiece.pc'\n",
      "    to the PKG_CONFIG_PATH environment variable\n",
      "    No package 'sentencepiece' found\n",
      "    Failed to find sentencepiece pkgconfig\n",
      "    ----------------------------------------\u001b[0m\n",
      "\u001b[33mWARNING: Discarding https://files.pythonhosted.org/packages/f8/81/297da1e97862684b3c3045ab83010e4a3b771e797a54a715742d189551a0/sentencepiece-0.0.2.tar.gz#sha256=9544a12d8db23490f9d9133d525f3175dafe0c5c10de5a878cbf3a026e8dbb85 (from https://pypi.org/simple/sentencepiece/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.\u001b[0m\n",
      "  Using cached sentencepiece-0.0.0.tar.gz (183 kB)\n",
      "\u001b[31m    ERROR: Command errored out with exit status 1:\n",
      "     command: /Users/filiphaltmayer/opt/miniconda3/envs/coinbase_demo/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '\"'\"'/private/var/folders/t1/4dsdtlv12r7cpxx9hkz33b840000gn/T/pip-install-pj4cikfj/sentencepiece_5ec4da9bd0674a07988d66a6db5b60bd/setup.py'\"'\"'; __file__='\"'\"'/private/var/folders/t1/4dsdtlv12r7cpxx9hkz33b840000gn/T/pip-install-pj4cikfj/sentencepiece_5ec4da9bd0674a07988d66a6db5b60bd/setup.py'\"'\"';f = getattr(tokenize, '\"'\"'open'\"'\"', open)(__file__) if os.path.exists(__file__) else io.StringIO('\"'\"'from setuptools import setup; setup()'\"'\"');code = f.read().replace('\"'\"'\\r\\n'\"'\"', '\"'\"'\\n'\"'\"');f.close();exec(compile(code, __file__, '\"'\"'exec'\"'\"'))' egg_info --egg-base /private/var/folders/t1/4dsdtlv12r7cpxx9hkz33b840000gn/T/pip-pip-egg-info-tgwxkrhi\n",
      "         cwd: /private/var/folders/t1/4dsdtlv12r7cpxx9hkz33b840000gn/T/pip-install-pj4cikfj/sentencepiece_5ec4da9bd0674a07988d66a6db5b60bd/\n",
      "    Complete output (11 lines):\n",
      "    Package sentencepiece was not found in the pkg-config search path.\n",
      "    Perhaps you should add the directory containing `sentencepiece.pc'\n",
      "    to the PKG_CONFIG_PATH environment variable\n",
      "    No package 'sentencepiece' found\n",
      "    Traceback (most recent call last):\n",
      "      File \"<string>\", line 1, in <module>\n",
      "      File \"/private/var/folders/t1/4dsdtlv12r7cpxx9hkz33b840000gn/T/pip-install-pj4cikfj/sentencepiece_5ec4da9bd0674a07988d66a6db5b60bd/setup.py\", line 28, in <module>\n",
      "        cmd('pkg-config sentencepiece --cflags'),\n",
      "      File \"/private/var/folders/t1/4dsdtlv12r7cpxx9hkz33b840000gn/T/pip-install-pj4cikfj/sentencepiece_5ec4da9bd0674a07988d66a6db5b60bd/setup.py\", line 14, in cmd\n",
      "        return os.popen(line).readlines()[0][:-1].split()\n",
      "    IndexError: list index out of range\n",
      "    ----------------------------------------\u001b[0m\n",
      "\u001b[33mWARNING: Discarding https://files.pythonhosted.org/packages/2e/d5/409b1f1cad792bdc5febbf2d6d4467f6e4a6d73672b74214385b33836ec1/sentencepiece-0.0.0.tar.gz#sha256=39ae6846b23460dca80b57260ba784b5de3af63d5dddf9e76abfc5ff24dc0696 (from https://pypi.org/simple/sentencepiece/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.\u001b[0m\n",
      "INFO: pip is looking at multiple versions of pymilvus to determine which version is compatible with other requirements. This could take a while.\n"
     ]
    }
   ],
   "source": [
    "! pip install -r requirements.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b2de997",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Starting Milvus Server\n",
    "\n",
    "This demo uses Milvus 1.1.0, please refer to the [Install Milvus](https://milvus.io/docs/v1.1.0/install_milvus.md) guide to learn how to use this docker container. For this example we wont be mapping any local volumes. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "c877e285",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "c24202156db5ee4dbe5a133d103dcd315313cbfb3e0c44bf26f34eaff8b50864\n"
     ]
    }
   ],
   "source": [
    "!docker run -d --name milvus_cpu_1.1.0 \\\n",
    "-p 19530:19530 \\\n",
    "-p 19121:19121 \\\n",
    "milvusdb/milvus:1.1.0-cpu-d050721-5e559c"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c2d62e7",
   "metadata": {},
   "source": [
    "### Starting Redis Server\n",
    "We are using Redis as a metadata storage service. Code can easily be modified to use a python dictionary, but that usually does not work in any use case outside of quick examples. We need a metadata storage service in order to be able to be able to map between embeddings and the corresponding data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "b4b9865e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "c6593309b1ef1998e32b0fd66174e3e93fe360800c12cfb8ab6c4797ff3a384b\n"
     ]
    }
   ],
   "source": [
    "!docker run  --name redis -d -p 6379:6379 redis"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e6f7c8da-ffa4-453c-bbcc-1275dfc4d444",
   "metadata": {},
   "source": [
    "### Confirm Running Servers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "6df36314-53de-437c-8ede-d8ac4431581b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "    __  _________ _   ____  ______    \n",
      "   /  |/  /  _/ /| | / / / / / __/    \n",
      "  / /|_/ // // /_| |/ / /_/ /\\ \\    \n",
      " /_/  /_/___/____/___/\\____/___/     \n",
      "\n",
      "Welcome to use Milvus!\n",
      "Milvus Release version: v1.1.0, built at 2021-05-06 14:50.43, with OpenBLAS library.\n",
      "You are using Milvus CPU edition\n",
      "Last commit id: 5e559cd7918297bcdb55985b80567cb6278074dd\n",
      "\n",
      "Loading configuration from: /var/lib/milvus/conf/server_config.yaml\n",
      "WARNNING: You are using SQLite as the meta data management, which can't be used in production. Please change it to MySQL!\n",
      "Supported CPU instruction sets: avx2, sse4_2\n",
      "FAISS hook AVX2\n",
      "Milvus server started successfully!\n"
     ]
    }
   ],
   "source": [
    "! docker logs milvus_cpu_1.1.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "7b81529f-e599-4627-b944-7f722b2f3605",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1:C 18 May 2021 19:36:22.376 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo\n",
      "1:C 18 May 2021 19:36:22.376 # Redis version=6.2.1, bits=64, commit=00000000, modified=0, pid=1, just started\n",
      "1:C 18 May 2021 19:36:22.377 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf\n",
      "1:M 18 May 2021 19:36:22.379 * monotonic clock: POSIX clock_gettime\n",
      "1:M 18 May 2021 19:36:22.381 * Running mode=standalone, port=6379.\n",
      "1:M 18 May 2021 19:36:22.381 # Server initialized\n",
      "1:M 18 May 2021 19:36:22.382 * Ready to accept connections\n"
     ]
    }
   ],
   "source": [
    "! docker logs redis"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cac14f13",
   "metadata": {},
   "source": [
    "### Downloading Pretrained Models\n",
    "This PaddlePaddle model is used to transform user information into vectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "b52f00c3",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2021-05-17 16:31:35--  https://paddlerec.bj.bcebos.com/aistudio/user_vector.tar.gz\n",
      "Resolving paddlerec.bj.bcebos.com (paddlerec.bj.bcebos.com)... 103.235.46.61\n",
      "Connecting to paddlerec.bj.bcebos.com (paddlerec.bj.bcebos.com)|103.235.46.61|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 33650924 (32M) [application/x-gzip]\n",
      "Saving to: ‘user_vector.tar.gz’\n",
      "\n",
      "user_vector.tar.gz  100%[===================>]  32.09M  5.40MB/s    in 17s     \n",
      "\n",
      "2021-05-17 16:31:54 (1.84 MB/s) - ‘user_vector.tar.gz’ saved [33650924/33650924]\n",
      "\n",
      "mkdir: movie_recommender/user_vector_model: File exists\n"
     ]
    }
   ],
   "source": [
    "!wget https://paddlerec.bj.bcebos.com/aistudio/user_vector.tar.gz --no-check-certificate\n",
    "!mkdir movie_recommender/user_vector_model\n",
    "!tar xf user_vector.tar.gz -C movie_recommender/user_vector_model/\n",
    "!rm user_vector.tar.gz"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ab3a252",
   "metadata": {},
   "source": [
    "### Downloading Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "39a7facb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2021-05-17 16:20:24--  https://paddlerec.bj.bcebos.com/aistudio/movies.dat\n",
      "Resolving paddlerec.bj.bcebos.com (paddlerec.bj.bcebos.com)... 103.235.46.61\n",
      "Connecting to paddlerec.bj.bcebos.com (paddlerec.bj.bcebos.com)|103.235.46.61|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 171308 (167K) [application/octet-stream]\n",
      "Saving to: ‘movie_recommender/movies.dat’\n",
      "\n",
      "movies.dat          100%[===================>] 167.29K  37.1KB/s    in 4.5s    \n",
      "\n",
      "2021-05-17 16:20:31 (37.1 KB/s) - ‘movie_recommender/movies.dat’ saved [171308/171308]\n",
      "\n",
      "--2021-05-17 16:20:31--  https://paddlerec.bj.bcebos.com/aistudio/movie_vectors.txt\n",
      "Resolving paddlerec.bj.bcebos.com (paddlerec.bj.bcebos.com)... 103.235.46.61\n",
      "Connecting to paddlerec.bj.bcebos.com (paddlerec.bj.bcebos.com)|103.235.46.61|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 1095505 (1.0M) [text/plain]\n",
      "Saving to: ‘movie_recommender/movie_vectors.txt’\n",
      "\n",
      "movie_vectors.txt   100%[===================>]   1.04M   271KB/s    in 4.0s    \n",
      "\n",
      "2021-05-17 16:20:36 (271 KB/s) - ‘movie_recommender/movie_vectors.txt’ saved [1095505/1095505]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Download movie information\n",
    "!wget -P movie_recommender https://paddlerec.bj.bcebos.com/aistudio/movies.dat --no-check-certificate\n",
    "# Download movie vecotrs\n",
    "!wget -P movie_recommender https://paddlerec.bj.bcebos.com/aistudio/movie_vectors.txt --no-check-certificate"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "090710b4",
   "metadata": {},
   "source": [
    "## Code Overview\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e994eb1e-aa76-446b-98c6-02c74f050ba5",
   "metadata": {},
   "source": [
    "### Importing Movies into Milvus"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a999eeb-bcc6-4800-9039-f9c57ea399f1",
   "metadata": {},
   "source": [
    "#### 1. Connectings to Milvus and Redis\n",
    "Both servers are running as Docker containers on the localhost with their corresponding default ports."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "d8de5e40",
   "metadata": {},
   "outputs": [],
   "source": [
    "from milvus import Milvus, IndexType, MetricType, Status\n",
    "import redis\n",
    "\n",
    "milv = Milvus(host = '127.0.0.1', port = 19530)\n",
    "r = redis.StrictRedis(host=\"127.0.0.1\", port=6379) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a3c114a7",
   "metadata": {},
   "source": [
    "#### 2. Loading Movies into Redis\n",
    "We begin by loading all the movie files into redis. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "f56cf19c",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import codecs\n",
    "\n",
    "#1::Toy Story (1995)::Animation|Children's|Comedy\n",
    "def process_movie(lines, redis_cli):\n",
    "    for line in lines:\n",
    "        if len(line.strip()) == 0:\n",
    "            continue\n",
    "        tmp = line.strip().split(\"::\")\n",
    "        movie_id = tmp[0]\n",
    "        title = tmp[1]\n",
    "        genre_group = tmp[2]\n",
    "        tmp = genre_group.strip().split(\"|\")\n",
    "        genre = tmp\n",
    "        movie_info = {\"movie_id\" : movie_id,\n",
    "                \"title\" : title,\n",
    "                \"genre\" : genre\n",
    "                }\n",
    "        redis_cli.set(\"{}##movie_info\".format(movie_id), json.dumps(movie_info))\n",
    "        \n",
    "with codecs.open(\"movie_recommender/movies.dat\", \"r\",encoding='utf-8',errors='ignore') as f:\n",
    "        lines = f.readlines()\n",
    "        process_movie(lines, r)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a54a6046",
   "metadata": {},
   "source": [
    "#### 3. Creating Partition and Collection in Milvus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "ef3ef1f7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Status(code=0, message='OK')"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "COLLECTION_NAME = 'demo_films'\n",
    "PARTITION_NAME = 'Movie'\n",
    "\n",
    "#Dropping collection for clean slate run\n",
    "milv.drop_collection(COLLECTION_NAME)\n",
    "\n",
    "\n",
    "param = {'collection_name':COLLECTION_NAME, \n",
    "         'dimension':32, \n",
    "         'index_file_size':2048, \n",
    "         'metric_type':MetricType.L2\n",
    "        }\n",
    "\n",
    "milv.create_collection(param)\n",
    "milv.create_partition(COLLECTION_NAME, PARTITION_NAME)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d298372e",
   "metadata": {},
   "source": [
    "#### 4. Getting Embeddings and IDs\n",
    "The vectors in `movie_vectors.txt` are obtained from the `user_vector_model` downloaded above. So we can directly get the vectors and the IDs by reading the file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "1aaee36b",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_vectors():\n",
    "    with codecs.open(\"movie_recommender/movie_vectors.txt\", \"r\", encoding='utf-8', errors='ignore') as f:\n",
    "        lines = f.readlines()\n",
    "    ids = [int(line.split(\":\")[0]) for line in lines]\n",
    "    embeddings = []\n",
    "    for line in lines:\n",
    "        line = line.strip().split(\":\")[1][1:-1]\n",
    "        str_nums = line.split(\",\")\n",
    "        emb = [float(x) for x in str_nums]\n",
    "        embeddings.append(emb)\n",
    "    return ids, embeddings\n",
    "\n",
    "ids, embeddings = get_vectors()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a6140b1",
   "metadata": {},
   "source": [
    "#### 4. Importing Vectors into Milvus\n",
    "Import vectors into the partition **Movie** under the collection **demo_films**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "4ac4cfff",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Status(code=0, message='Add vectors successfully!')"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "status = milv.insert(collection_name=COLLECTION_NAME, records=embeddings, ids=ids, partition_tag=PARTITION_NAME)\n",
    "status[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e93feb30",
   "metadata": {},
   "source": [
    "### Recalling Vectors in Milvus\n",
    "#### 1. Genarating User Embeddings\n",
    "Pass in the gender, age and occupation of the user we want to recommend. **user_vector_model** model will generate the corresponding user vector.\n",
    "Occupation is chosen from the following choices:\n",
    "*  0:  \"other\" or not specified\n",
    "*  1:  \"academic/educator\"\n",
    "*  2:  \"artist\"\n",
    "*  3:  \"clerical/admin\"\n",
    "*  4:  \"college/grad student\"\n",
    "*  5:  \"customer service\"\n",
    "*  6:  \"doctor/health care\"\n",
    "*  7:  \"executive/managerial\"\n",
    "*  8:  \"farmer\"\n",
    "*  9:  \"homemaker\"\n",
    "*  10:  \"K-12 student\"\n",
    "*  11:  \"lawyer\"\n",
    "*  12:  \"programmer\"\n",
    "*  13:  \"retired\"\n",
    "*  14:  \"sales/marketing\"\n",
    "*  15:  \"scientist\"\n",
    "*  16:  \"self-employed\"\n",
    "*  17:  \"technician/engineer\"\n",
    "*  18:  \"tradesman/craftsman\"\n",
    "*  19:  \"unemployed\"\n",
    "*  20:  \"writer\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "1a35a9d4",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2021-05-18 13:06:14,131 - INFO - load_model_config params: model_path:movie_recommender/user_vector_model/serving_server_dir, use_gpu:False,            gpu_id:0, use_profile:False, thread_num:1, mem_optim:True, ir_optim:False,            use_trt:False, use_feed_fetch_ops:False\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "from paddle_serving_app.local_predict import LocalPredictor\n",
    "\n",
    "class RecallServerServicer(object):\n",
    "    def __init__(self):\n",
    "        self.uv_client = LocalPredictor()\n",
    "        self.uv_client.load_model_config(\"movie_recommender/user_vector_model/serving_server_dir\") \n",
    "        \n",
    "    def hash2(self, a):\n",
    "        return hash(a) % 1000000\n",
    "\n",
    "    def get_user_vector(self):\n",
    "        dic = {\"userid\": [], \"gender\": [], \"age\": [], \"occupation\": []}\n",
    "        lod = [0]\n",
    "        dic[\"userid\"].append(self.hash2('0'))\n",
    "        dic[\"gender\"].append(self.hash2('M'))\n",
    "        dic[\"age\"].append(self.hash2('23'))\n",
    "        dic[\"occupation\"].append(self.hash2('6'))\n",
    "        lod.append(1)\n",
    "\n",
    "        dic[\"userid.lod\"] = lod\n",
    "        dic[\"gender.lod\"] = lod\n",
    "        dic[\"age.lod\"] = lod\n",
    "        dic[\"occupation.lod\"] = lod\n",
    "        for key in dic:\n",
    "            dic[key] = np.array(dic[key]).astype(np.int64).reshape(len(dic[key]),1)\n",
    "        fetch_map = self.uv_client.predict(feed=dic, fetch=[\"save_infer_model/scale_0.tmp_1\"], batch=True)\n",
    "        return fetch_map[\"save_infer_model/scale_0.tmp_1\"].tolist()[0]\n",
    "\n",
    "recall = RecallServerServicer()\n",
    "user_vector = recall.get_user_vector()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e15ea6e8",
   "metadata": {
    "tags": []
   },
   "source": [
    "#### 2. Searching\n",
    "Pass in the user vector, and then recall vectors in the previously imported data collection and partition."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "e4d91d02",
   "metadata": {},
   "outputs": [],
   "source": [
    "TOP_K = 20\n",
    "SEARCH_PARAM = {'nprobe': 20}\n",
    "status, results = milv.search(collection_name=COLLECTION_NAME, query_records=[user_vector], top_k=TOP_K, params=SEARCH_PARAM)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c847608",
   "metadata": {},
   "source": [
    "#### 3. Returning Information by IDs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "90a56325",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['{\"movie_id\": \"760\", \"title\": \"Stalingrad (1993)\", \"genre\": [\"War\"]}',\n",
       " '{\"movie_id\": \"1350\", \"title\": \"Omen, The (1976)\", \"genre\": [\"Horror\"]}',\n",
       " '{\"movie_id\": \"1258\", \"title\": \"Shining, The (1980)\", \"genre\": [\"Horror\"]}',\n",
       " '{\"movie_id\": \"632\", \"title\": \"Land and Freedom (Tierra y libertad) (1995)\", \"genre\": [\"War\"]}',\n",
       " '{\"movie_id\": \"3007\", \"title\": \"American Movie (1999)\", \"genre\": [\"Documentary\"]}',\n",
       " '{\"movie_id\": \"2086\", \"title\": \"One Magic Christmas (1985)\", \"genre\": [\"Drama\", \"Fantasy\"]}',\n",
       " '{\"movie_id\": \"1051\", \"title\": \"Trees Lounge (1996)\", \"genre\": [\"Drama\"]}',\n",
       " '{\"movie_id\": \"3920\", \"title\": \"Faraway, So Close (In Weiter Ferne, So Nah!) (1993)\", \"genre\": [\"Drama\", \"Fantasy\"]}',\n",
       " '{\"movie_id\": \"1303\", \"title\": \"Man Who Would Be King, The (1975)\", \"genre\": [\"Adventure\"]}',\n",
       " '{\"movie_id\": \"652\", \"title\": \"301, 302 (1995)\", \"genre\": [\"Mystery\"]}',\n",
       " '{\"movie_id\": \"1605\", \"title\": \"Excess Baggage (1997)\", \"genre\": [\"Adventure\", \"Romance\"]}',\n",
       " '{\"movie_id\": \"1275\", \"title\": \"Highlander (1986)\", \"genre\": [\"Action\", \"Adventure\"]}',\n",
       " '{\"movie_id\": \"1126\", \"title\": \"Drop Dead Fred (1991)\", \"genre\": [\"Comedy\", \"Fantasy\"]}',\n",
       " '{\"movie_id\": \"792\", \"title\": \"Hungarian Fairy Tale, A (1987)\", \"genre\": [\"Fantasy\"]}',\n",
       " '{\"movie_id\": \"2228\", \"title\": \"Mountain Eagle, The (1926)\", \"genre\": [\"Drama\"]}',\n",
       " '{\"movie_id\": \"2659\", \"title\": \"It Came from Hollywood (1982)\", \"genre\": [\"Comedy\", \"Documentary\"]}',\n",
       " '{\"movie_id\": \"2545\", \"title\": \"Relax... It\\'s Just Sex (1998)\", \"genre\": [\"Comedy\"]}',\n",
       " '{\"movie_id\": \"1289\", \"title\": \"Koyaanisqatsi (1983)\", \"genre\": [\"Documentary\", \"War\"]}',\n",
       " '{\"movie_id\": \"2537\", \"title\": \"Beyond the Poseidon Adventure (1979)\", \"genre\": [\"Adventure\"]}',\n",
       " '{\"movie_id\": \"2864\", \"title\": \"Splendor (1999)\", \"genre\": [\"Comedy\"]}']"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "recall_results = []\n",
    "for x in results[0]:\n",
    "    recall_results.append(r.get(\"{}##movie_info\".format(x.id)).decode('utf-8'))\n",
    "recall_results"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4f7e3c5",
   "metadata": {},
   "source": [
    "## Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "843120ee",
   "metadata": {},
   "source": [
    "After completing the recall service, the results can be further sorted using the **movie_recommender** model, and then the movies with high similarity scores can be recommended to users. You can try this deployable recommendation system using this [quick start](QUICK_START.md)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7c9137a2-14bf-48fb-8173-0c7afbbbc3a3",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
